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Abstract 

Organizations  that  adapt  to  changes  in  their  environment  have  a  much  better  chance  at 
achieving  their  mission  and  performing  well.  When  organizations  do  adapt  they  usually 
alter  their  strategies,  but  rarely  do  they  change  their  organizational  structure  even  when 
that  structure  no  longer  fits  the  mission.  To  identify  the  conditions  that  will  be  salient 
enough  to  cause  organizations  to  alter  not  only  their  strategies,  but  also  their  current 
structures,  we  used  a  model-based  design  process  to  create  mission  scenarios  that  were 
either  matched  (congruent)  or  mismatched  (incongruent)  with  two  organizational 
structures  (functional,  divisional).  We  then  examined  measures  over  time  to  identify 
ones  that  discriminated  between  congruent  and  incongruent  conditions  early  in  the 
mission  scenarios.  Several  measures  of  communications,  performance  and  workload 
appeared  to  be  good  candidates  for  leading  indicators  of  the  need  for  organizational 
change. 


Introduction 

It  is  well  accepted  that  organizations  that  can  adapt  to  internal  changes  or  changes  in  their 
environment  have  a  much  better  chance  at  achieving  their  mission  and  performing  well. 
Indeed,  it  has  been  demonstrated  that  high  performing  organizations  are  adept  at  sensing 
changes  in  their  environment  and  altering  their  strategies  to  accommodate  these  changes 
without  compromising  performance  (Entin  &  Serfaty,  1999).  Little,  however,  is  known 
about  the  dynamics  of  change  or  the  conditions  that  might  lead  organizations  to  alter  their 
strategies  or  make  changes  to  their  current  organizational  structures.  Moreover,  while 
there  is  evidence  that  organizations  will  institute  strategy  changes,  they  appear  quite 
reluctant  to  make  changes  to  their  structures  (Entin,  Serfaty,  and  Kerrigan,  1998; 
Hollenbeck  et  ah,  1999). 

To  better  understand  and  identify  the  conditions  that  will  cause  organizations  to  alter  not 
only  their  strategies,  but  also  their  current  structures,  we  need  to  know  more  about  ways 
to  facilitate  structural  adaptation.  In  other  words,  what  are  the  diagnostic  conditions,  or 
leading  indicators,  that  organizations  could  use  to  realize  that  their  organizational 
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structures  no  longer  fit  particular  missions  (incongruence)?  And  at  what  point  does  the 
“pain”  of  incongruence  become  so  great  that  structural  change  is  initiated?  The  purpose 
of  this  presentation  is  to  explore  the  first  question  in  order  to  identify  leading  indicators 
of  the  need  to  change.  This  work  will  set  the  stage  for  future  empirical  evaluations  of 
structural  adaptation  in  action. 

The  present  paper  builds  on  the  manipulation  of  organizational  congruence  presented  by 
Diedrich  et  al.  (2003,  this  volume).  The  approach  used  by  Diedrich  et  al.  was  to  define 
two  disparate  organizational  structures  and  then  design  two  missions  (scenarios)  that 
exploited  the  differences  between  the  two  structures.  Thus,  the  objective  was  for  the  first 
mission  scenario  to  be  “tuned”  to  organization  1  through  a  high  degree  of  congruence, 
while  also  being  “mismatched”  (i.e.,  exhibit  low  congruence)  with  organization  2.  The 
goal  was  for  the  reverse  to  be  true  for  the  second  scenario.  Thus,  it  was  possible  to 
observe  an  organization  in  both  a  mismatched  (incongruent)  and  matched  (congruent) 
situation. 

Based  on  previous  work  (Diedrich,  Hocevar,  Entin,  Hutchins,  Kemple,  &  Kleinman, 
2002;  Hutchins,  Kleinman,  Hocevar,  Kemple,  &  Porter,  2001;  Moon,  Hollenbeck,  Ilgen, 
West,  Ellis,  Humphrey,  &  Porter,  2000),  the  two  organizational  structures  that  Diedrich 
et  al.  (2003)  explored  are  commonly  referred  to  as  functional  (F)  and  divisional  (D).  In 
this  case,  the  functional  structure  was  organized  such  that  each  participant  specialized  in 
one  or  two  aspects  of  a  mission  such  as  strike  using  assets  that  were  distributed  across 
multiple  platforms  (ships).  In  contrast,  in  the  divisional  structure,  each  participant  had 
control  over  a  multifunctional  platform  that  to  a  large  extent  was  able  to  process  a  variety 
of  functional  tasks.  Given  these  organizations,  and  based  on  the  modeling  supporting  this 
effort  (e.g.,  Kleinman  et  al.,  2003;  Levchuk  et  al.,  2002,  parts  I  &II;  Levchuk  et  al.,  2001) 
it  was  expected  due  to  the  greater  coordination  requirements  for  task  processing, 
communications  and  perceived  workload  would  increase  in  the  incongruent  conditions. 
Likewise,  given  the  additional  coordination  and  communication  requirements,  it  was 
predicted  that  performance  would  suffer. 

Overall,  the  findings  reported  by  Diedrich  et  al.  (2003)  were  consistent  with  these  model- 
based  predictions.  These  authors  explored  the  F  and  D  organizations  in  two  scenarios, 
one  of  which  was  functional-favoring  (f)  and  one  of  which  was  divisional  favoring  (d). 
Relative  to  the  congruent  cases  (F-f,  D-d),  communications  increased  in  the  mismatched 
conditions  (F-d,  D-f),  reflecting  the  model-based  manipulation  of  the  need  for  cross¬ 
participant  coordination,  which  was  the  primary  tool  employed  to  engineer  congruence. 
Similarly,  consistent  with  these  manipulations,  the  perceived  workload  also  tended  to 
increase,  while  the  level  of  performance  tended  to  decrease  in  the  mismatched  conditions. 
Collectively,  these  results  provided  strong  evidence  that  the  congruence  manipulation 
was  effective. 

Based  on  these  data,  Diedrich  et  al.  (2003)  also  suggested  that  communication  strategies 
might  provide  suitable  “leading  indicators”  of  the  need  for  structural  change.  Leading 
indicators  are  defined  as  measures  that  signal  that  an  organization’s  structure  is  no  longer 
congruent  with  its  mission  and  that  change  is  needed  to  optimize  mission  effectiveness. 
Strategies  are  interesting  potential  leading  indicators  because,  when  teams  sense 
deterioration  in  their  performance  due  to  stress,  increased  workload,  or  other  conditions, 
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they  tend  to  attempt  to  ameliorate  these  problems  by  adopting  different  strategies  (Entin 
and  Serfaty,  1999).  In  particular,  communication  strategies  are  important  because 
communications  are  the  vehicle  through  which  teams  attempt  to  deal  collectively  with 
mission  execution.  Consequently,  Diedrich  et  al.  (2003)  argued  that  these  strategy 
changes  might  offer  insight  into  leading  indicators  of  the  need  for  change  to  the  extent 
that  they  reliably  reflect  how  an  organization  attempts  to  cope  with  the  difficulties 
associated  with  incongruence.  Consistent  with  this  claim,  results  indicated  that 
communication  strategies  did  indeed  change  in  the  face  of  incongruence,  but  that  the 
specific  changes  observed  depended  on  the  context  provided  by  the  organizational 
structures  and  mission  scenarios. 

However,  these  findings  were  based  on  measures  taken  over  the  course  of  the  entire 
mission  scenarios.  While  interesting,  ideally  speaking,  leading  indicators  should  predict 
the  need  for  change  early  in  a  mission  prior  to  any  severe  performance  decrements. 
Accordingly,  in  this  paper  we  explored  potential  leading  indicators  by  concentrating  on 
measures  taken  over  time.  Similar  to  Diedrich  et  al.  (2003),  we  once  again  concentrated 
on  measures  of  communication,  workload,  and  performance.  We  believe  that  by 
examining  measures  taken  over  time,  we  can  identify  if  and  when  the  measures  are 
sensitive  to  deteriorating  conditions,  thus  shedding  light  on  what  measures  might  make 
good  candidates  as  leading  indicators. 


Method 


Participants 

Forty-eight  officers  attending  the  Naval  Postgraduate  School  served  as  participants.  Most 
of  the  officers  were  03  or  04  and  several  services  and  nations  were  represented. 
Participants  were  organized  into  eight  teams  of  six  individuals  each.  For  complete  details 
on  scenario  design,  organizational  design,  and  method,  see  Diedrich  et  al.  (2003)  and 
Kleinman  et  al.  (2003). 

Experimental  Design  and  Independent  Variables 

There  were  two  primary  independent  variables.  The  first,  organizational  structure,  was 
manipulated  as  a  between-subjects  variable  and  was  comprised  of  two  levels  (divisional 
(D)  and  functional  (F)).  Each  team  performed  in  only  F  or  D.  In  the  divisional 
organizational  structure,  team  members  “owned”  a  multifunctional  platform  (ship)  that 
they  used  to  process  air,  surface,  and  subsurface  threats  in  a  specified  geographical  area. 
For  the  functional  organizational  structure,  each  team  member  specialized  in  one  or  two 
aspect  of  the  mission  such  as  air  warfare  or  ballistic  missile  defense.  The  assets 
necessary  for  a  particular  warfare  area  resided  on  various  platforms  distributed  over  the 
entire  battle  space. 

The  second  primary  independent  variable  was  scenario,  which  was  manipulated  as  a 
within-subjects  factor  with  two  levels  (functional  (f)  and  divisional  (d)).  Each  team 
performed  in  both  d  and  f  across  two  replications  (fddf  or  dffd).  In  brief,  two  scenarios 
were  developed,  one  that  was  congruent  with  the  divisional  organizational  structure  (d) 
and  one  that  was  congruent  with  the  functional  organizational  structure  (f).  Thus,  by 
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manipulating  the  presentation  of  the  two  scenario  types  across  the  two  organizational 
structures,  we  created  conditions  of  congruence  (F-f  and  D-d)  and  incongruence  (F-d  and 
D-f). 


Simulator  and  Scenario 

The  simulation  was  implemented  in  the  Distributed  Dynamic  Decision-making  (DDD) 
environment.  The  DDD  is  a  distributed  client  server  simulation  that  provides  a  flexible 
framework  in  which  to  study  individual  and  team  performance.  DDD  simulations 
involve  individual  (and  team)  decision-making  about  complex  situations  based  on 
information  and  resources  provided  by  the  simulation  (and  other  team  members)  (Serfaty 
&  Kleinman,  1985;  Kleinman  &  Serfaty,  1989).  As  presented  in  Kleinman  et  al.  (2003), 
the  F  and  D  organizations  and  f  and  d  scenarios  were  implemented  in  the  DDD.  To 
manipulate  congruence  or  incongruence,  several  variables  were  employed  including 
between-team  member  coordination  requirements  for  processing  mission  tasks  and  task 
phasing  (i.e.,  waves  of  attacks  directed  so  as  to  overwhelm  particular  team  members 
based  on  their  mission  roles). 

Procedure 

Participants  began  the  experiment  by  first  signing  a  standard  consent  form  and  then 
completing  two  hours  of  DDD  “buttonology”  training  to  learn  how  to  control  the  various 
assets  and  how  to  use  the  various  functions  contained  in  the  DDD  simulation.  A  second 
two-hour  session  provided  training  designed  to  provide  the  skills  necessary  to  perform 
either  the  divisional  or  functional  scenarios,  without  creating  a  bias  for  either  kind  of 
scenario.  Teams  next  engaged  in  a  two-hour  data  collection  session,  where  in  a  counter¬ 
balanced  order,  they  performed  a  divisional  and  functional  scenario.  A  second  two-hour 
data  collection  session  followed  anywhere  from  several  hours  to  several  days  later.  Once 
again,  teams  completed  a  divisional  and  functional  scenario.  In  general,  the  mission 
involved  preparing  a  battlespace  for  the  arrival  of  follow-on  forces  by  engaging  hostile 
assets  perceived  as  immediate  threats.  Simultaneously,  players  were  charged  with 
destroying  or  capturing  an  enemy  command  post,  two  enemy  air  bases,  two  enemy  naval 
bases,  and  a  port.  In  addition,  players  had  to  defend  neighboring  foreign  friendly  areas 
from  SCUD  missile  attacks.  For  complete  details  on  the  methodologies  used  see 
Diedrich  et  al.  (2003). 

Data  Analysis 

Dependent  measures  were  derived  from  three  different  sources:  instruments  completed  by 
trained  observers,  participant  self-report  measures,  and  measures  derived  from  the  DDD 
simulator.  Below,  we  briefly  describe  the  instruments  employed  for  each  of  these  areas 
in  context  of  our  emphasis  on  measures  over  time. 

Team  Communications.  Verbal  communications  among  the  members  of  the  team  were 
captured  by  two  observers  at  an  intermediate  level  of  detail  that  incorporated  both 
semantic  and  quantitative  aspects  of  the  communication  stream  (Entin,  1999).  During  the 
conduct  of  a  scenario  the  observers  listened  to  the  communications  in  real  time,  and  used 
specially  design  software  and  a  handheld,  touch-sensitive  screen  computer  to  code  the 
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source,  recipient,  time,  and  type  of  the  verbal  communications  among  the  team  members. 
One  of  the  observers  coded  the  communications  for  three  of  the  team  members,  while  the 
other  observer  coded  the  communications  exhibited  by  the  remaining  team  members. 
Types  of  communications  were  divided  into  three  basic  categories:  transfers,  requests, 
and  acknowledgements.  Both  transfers  and  requests  were,  in  turn,  classified  as  requests 
for  or  transfer  of  information,  action,  or  coordination. 

Workload.  At  five-minute  intervals  throughout  the  scenario  the  DDD  simulator  prompted 
each  team  member  to  provide  an  estimate  of  the  workload  they  were  experiencing  at  the 
moment.  The  prompt  was  in  the  form  of  a  window  that  opened  on  the  display.  The 
workload  rating  was  made  on  a  seven  point  Likert  scale  anchored  at  one  end  by  the  word 
“low”  and  the  other  end  by  “high.”  The  windows,  which  were  mildly  intrusive,  could  not 
be  closed  until  a  rating  was  made. 

Performance  Measures.  The  DDD  simulator  provided  several  measures  of  performance 
over  time.  These  measures  included  the  number  of  tasks  to  be  processed  within  a  given 
time  period,  the  number  of  tasks  processed  within  a  given  time  period  (attacks),  the  time 
at  which  a  task  was  processed,  and  a  measure  of  accuracy  of  processing  (resources 
applied/resources  needed). 


Results 

Given  our  emphasis  on  leading  indicators  of  the  need  to  change,  in  this  paper  we 
analyzed  our  data  over  time,  contrasting  the  congruent  cases  (D-d,  F-f)  with  the 
incongruent  cases  (D-f,  F-d).  The  goal  of  these  analyses  was  to  identify  those  variables 
that  showed  early  signs  of  differences  that  could  be  used  to  signal  the  need  for  change. 
Our  analyses  focused  on  the  same  classes  of  measures  described  by  Diedrich  et  al. 
(2003),  including  performance  over  time,  communications  over  time,  and  workload  over 
time.  It  should  be  noted  that,  in  general,  the  differences  between  the  congruent  and 
incongruent  conditions  for  each  of  measures  listed  above  were  all  found  to  be  significant 
when  analyzed  as  end  of  trial  measures  (Diedrich  et  al.,  2003).  Building  on  these 
analyses,  the  measures  for  these  variables  over  time  also  generated  significant  differences 
between  the  congruent  and  incongruent  conditions  over  the  time  periods. 

Tempo 

We  began  these  analyses  by  first  deriving  a  measure  of  the  tempo  of  the  scenarios.  In 
general,  scenario  tempo  was  created  by  the  rate  of  delivery  of  the  tasks  into  the  game  and 
the  rate  at  which  the  tasks  were  processed  or  disappeared  (i.e.,  timed  out).  Consequently, 
the  tempo  measure  reflects  the  number  of  tasks  on  the  screen  to  be  processed.  We 
present  this  measure  first  as  we  used  it  as  a  guide  to  understand  how  various  measures 
(communications,  performance,  workload)  might  unfold  over  time.  Results  are  shown  in 
Figure  1,  which  plots  time  period  by  the  maximum  number  of  tasks  to  be  processed.  The 
time  periods  represent  seven  5-minute  periods  that  covered  the  entire  scenario.  These 
data  show  that  in  general,  the  average  maximum  number  of  tasks  to  be  processed  peaked 
in  the  middle  of  the  scenario  runs.  This  curvilinear  pattern  is  also  reflected  in  the  various 
performance  and  process  measures  presented  next. 
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Figure  1.  Scenario  tempo  (maximum  number  of  tasks  to  be  processed). 


Communications  Over  Time 


As  noted  above,  the  model-based  manipulation  of  congruence  hinged  on  coordination 
requirements,  in  that  relative  to  the  congruent  conditions,  performance  in  the  incongruent 
conditions  required  much  greater  between-participant  coordination  to  complete  tasks 
accurately  (Diedrich  et  ah,  2003;  Kleinman  et  al.,  2003).  Consequently,  we  predicted 
that  communication  volume  would  be  enhanced  in  the  mismatched  conditions,  as  in 
general,  a  great  deal  of  coordination  is  achieved  through  verbal  exchange  within  the 
context  of  these  types  of  mission  scenarios  (Entin  and  Serfaty,  1999).  Consistent  with 
results  for  the  entire  scenario  (Diedrich  et  al.,  2003),  there  were  strong  and  consistent 
difference  between  conditions  early  in  the  scenarios.  Figure  2  shows  the  between 
condition  differences  across  seven  5-minute  time  periods.  For  the  F  organizations,  across 
the  entire  mission  scenario,  communication  volume  in  the  incongruent  case  (F-f)  greatly 
surpassed  that  seen  in  the  congruent  case  (F-d).  Similarly,  for  the  D  organization,  with 
the  exception  of  time  period  5,  communication  volume  in  the  D-f  case  surpassed  that  in 
the  D-d  case.  In  general,  however,  these  latter  differences  were  smaller  than  those  seen 
in  the  F  organization.  Collectively,  these  data  show  that  early  in  the  scenario,  differences 
in  communications  volume  can  potentially  provide  clues  to  incongruence.  However,  the 
magnitude  of  these  differences  depended  on  the  specific  scenario-structure  pairings. 


— D_d 
— D_f 
—a —  F_d 
F_f 


Figure  2.  Total  communication  frequency  for  each  congruency  condition. 
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Similar  to  the  overall  communications  volume,  Figure  3  shows  that  the  requests  for 
coordination  (e.g.,  “I  need  your  help...”)  and  statements  that  coordination  will,  is,  or  has 
occurred  (transfers,  e.g.,  “Ready  to  coordinate  on  ...”)  are  also  higher  in  the  incongruent 
than  the  congruent  cases  for  almost  all  the  time  periods.  Once  again,  these  data  are 
important  in  that  they  reflect  the  model-based  manipulation  of  coordination  requirements. 
In  response  to  this  manipulation,  not  only  were  communications  in  general  increased,  but 
likewise,  communications  specifically  about  coordination  increased.  Once  again,  these 
differences  were  present  early  in  the  scenarios. 


congruency  condition 

In  total,  the  communications  data  revealed  that  there  were  robust  differences  between  the 
congruent  and  incongruent  cases  early  in  the  scenarios.  These  differences  were 
particularly  profound  for  the  F  organization,  and  in  addition,  these  differences  appeared 
in  both  the  overall  communications  and  in  communications  specifically  about 
coordination.  These  data  suggest  that  communications  might  provide  early,  leading, 
indicators  of  the  need  for  organizational  change.  However,  the  precise  patterns  depended 
on  context  of  the  organizations  and  scenarios  (see  also,  Diedrich  et  ah,  2003). 

Workload  Over  Time 

Since  the  manipulation  of  congruency  hinged  on  coordination  requirements,  and  since 
more  coordination  and  communication  is  required  to  effectively  process  tasks  in  the 
incongruent  conditions,  we  predicated  that  there  would  be  an  increase  in  perceived 
workload  in  the  incongruent  conditions.  Over  the  course  of  the  entire  scenario,  results 
tended  to  show  higher  perceived  workloads  in  the  incongruent  conditions  for  both 
organizations  D  and  F.  Moreover,  as  shown  in  Figure  4,  there  were  also  differences  in 
perceived  workload  over  time.  However,  it  is  also  clear  that  the  patterns  across  the  seven 
time  periods  are  quite  varied  for  the  four  conditions.  Average  workload  scores  for  the 
two  functional  organization  conditions  showed  a  sharp  drop  in  time  period  7.  Teams  in 
the  divisional  organizational  structure  did  not  exhibit  this  pattern.  In  addition,  Figure  4 
shows  that  perceived  workload  for  the  four  congruency  conditions  to  be  quite  similar 
initially  and  then  start  to  diverge  in  time  period  three.  This  suggests  that,  although 
differences  in  perceived  workload  across  congruence  conditions  were  present  early,  these 
differences  were  not  present  as  early  in  the  scenario  as  the  communication  differences. 
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Figure  4.  Self-reported  workload  aggregated  by  team  and  plotted  over  time. 


Performance  over  time 


Based  on  the  enhanced  coordination  requirements,  we  expected  that  performance  would 
be  negatively  impacted  in  incongruent  conditions  due  to  an  inability  to  quickly  and 
accurately  process  a  large  number  of  tasks.  Indeed,  this  was  true  in  terms  of  overall 
performance  measures  with  respect  to  the  percent  of  task  processed  (Diedrich  et  al., 
2003).  Building  on  these  findings  and  to  explore  these  data  over  time,  we  derived  the 
number  of  attacks  initiated  by  teams  in  each  congruency  condition.  Once  again,  the  mean 
number  of  attacks  initiated  was  computed  for  each  five-minute  interval  of  the  scenario 
and  plotted  by  condition.  Figure  5  shows  that  in  terms  of  the  number  of  tasks  processed, 
the  two  congruent  conditions,  D-d  and  F-f,  performed  at  a  higher  level  than  their 
corresponding  incongruent  conditions,  D-f  and  F-d,  across  all  of  the  time  periods. 
Moreover,  the  differences  between  the  congruent  and  incongruent  conditions  appeared 
right  from  the  first  time  period  and  persisted  over  the  seven  time  periods. 


Figure  5.  Frequency  of  attacks  (tasks  processed)  over  time. 


To  more  fully  explore  performance  over  time,  we  then  calculated  a  temporal  performance 
measure  that  combined  accuracy,  timeliness  and  efficiency,  plus  task  value,  into  a  single 
metric  and  plotted  it  for  a  given  mission  scenario.  Task  value  was  included  as  it  reflects 
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the  importance  of  the  task  relative  to  the  commander’s  intent  or  priority.  According  to 
commander’s  intent,  all  tasks  were  not  equally  important  (Kleinman  et  al.,  2003). 


c 


Figure  6.  Accrued  gain  metric. 

In  order  to  understand  the  temporal  measure,  refer  to  Figure  6.  Assume  that  task  i  with 
value  Vi  is  competed  at  time  ti  with  accuracy  ACCi.  (Note:  different  formulae  might  be 
used  in  computing  ACCi  depending  on  the  experimental  context.)  The  “gain”  that  the 
team  accrues  at  time  ti  is  defined  as: 

Gi  =  Vi  *  ACCi 

Thus,  the  accumulated  gain  at  time  t,  G(t)  is  the  summation  over  all  tasks  i  completed  at 
or  prior  to  time  t: 


G(t)  =  Gi  *  u(t  -  ti) 

where  the  unit  step  function,  u(x)  =  1  if  x  >  0,  otherwise  u(x)  =  0.  The  plot  of  G(t)  versus 
time  t  is  thus  the  staircase  function  as  shown  in  Figure  6. 

In  general,  the  accumulated  gain  G(t)  has  several  desirable  properties  that  are 
immediately  evident.  If  tasks  are  processed  sooner,  the  rise  in  G(t)  is  earlier.  If  tasks  are 
processed  more  accurately,  the  rise  in  G(t)  is  steeper.  (Note  that  the  rise  is  also  a  function 
of  processed  task  value.)  A  comparison  of  two  gain  plots  would  thus  show  differences  in 
timeliness  or  accuracy  or  both.  In  addition,  the  area  under  the  accumulated  gain  plot  - 
either  as  an  accrued  area  or  a  total  area  to  the  end  of  the  scenario  -  provides  an  interesting 
aggregated  measure  of  performance  that  subsumes  a  number  of  individual  dimensions  as 
noted  above.  This  area  from  time  0  to  time  T  is  given  by: 
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Area  =  Gi  *  (T  -  ti)*u(T  -  ti) 


Applying  this  measure  to  congruent  and  incongruent  cases,  the  results  for  the  divisional 
scenario  were  45.8  (D-d)  and  33.5  (F-d).  For  the  functional  scenario,  the  results  were 
42.8  (F-f)  and  33.6  (D-f).  The  larger  areas  under  the  congruent  condition  curves  and  the 
smaller  area  under  the  incongruent  conditions  show  that  the  teams  were  less  timely  and 
accurate  processing  tasks  in  the  incongruent  conditions.  Figure  7  illustrates  the  over  time 
results.  Once  again,  from  early  in  the  mission  scenarios,  performance  differences  were 
present  that  could  serve  as  leading  indicators  of  the  need  for  structural  change. 


Figure  7.  Temporal  performance  accrued  gain  measure  plotted  for  the  four  congruence 

conditions. 

Discussion 

The  goal  of  the  over  time  analyses  reported  here  was  to  identify  candidate  leading 
indicators  that  signal  that  an  organization’s  structure  is  incongruent  with  its  mission. 
Consequently,  we  examined  measures  over  time  to  find  ones  that  were  sensitive  to  the 
incongruence  early  in  the  missions  before  performance  decrements  become  overly 
problematic.  In  the  main,  our  analyses  were  successful,  as  we  identified  several  over 
time  measures  that  have  promise  as  leading  indicators. 

Our  results  showed  that  measures  of  overall  communication  frequency  and 
communications  specifically  about  coordination  exhibited  the  differences  early  in  the 
mission  scenarios  and  that  the  differences  generally  persisted  over  the  mission  scenarios. 
Based  on  the  modeling  that  created  the  congruent  and  incongruent  conditions,  we 
expected  the  observed  increases.  However,  it  remained  to  be  empirically  determined  how 
early  these  difference  would  appear  and  how  persistent  they  would  be  over  the  scenarios. 
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Thus,  our  findings  were  particularly  noteworthy  and  satisfying.  Similarly,  the  finding 
that  perceived  workload  showed  differences  between  the  congruency  conditions  early  in 
the  mission  scenarios  also  suggested  that  self-reported  workload  may  qualify  as  a  leading 
indicator.  However,  the  workload  measure  seemed  to  lag  behind  the  observed 
differences  in  communication  strategies. 

Results  also  indicated  that  the  temporal  performances  measures  were  compelling.  In 
particular,  the  accrued  gain  measure  was  interesting  because  it  is  a  multidimensional 
measure  that  combines  task  accuracy,  timeliness,  and  task  value  (i.e.,  importance  or 
criticality)  into  a  single  easily  displayed  measure.  There  was,  however,  some  question  as 
to  how  sensitive  the  measure  was  to  incongruence  early  in  the  missions.  The  lower 
temporal  performance  was  revealed  early  in  the  scenario  for  the  F-d  incongruent 
condition,  but  this  was  not  the  case  for  the  D-f  incongruent  condition.  For  the  D-f 
condition,  over  10  minutes  (one-third)  of  the  scenario  went  by  before  the  drop  in 
temporal  performance  became  obvious.  On  the  other  hand,  performance  had  not 
significantly  deteriorated  up  to  that  point,  compared  to  the  congruent  condition. 
Nevertheless,  similar  to  workload,  this  measure  seemed  to  lag  behind  the  communication 
measures. 

Consequently,  although  each  of  the  measures  explored  here  showed  robust  and  promising 
differences,  within  the  context  studied  here,  in  general  the  communication  differences 
seemed  to  lead  the  other  measures.  Thus,  the  communication  measures  appeared  to  be 
particularly  promising  candidates  for  leading  indicators.  However,  Diedrich  et  al.  (2003) 
noted  that  the  precise  communication  differences  observed  depended  on  the  contexts 
created  by  the  structures  and  the  missions.  Similarly,  when  measured  over  time,  teams  in 
the  functional  organizational  structure  drastically  change  their  communication  patterns 
compared  to  teams  in  the  divisional  organizational  structure.  These  findings  suggest  that 
it  is  likely  that  as  additional  conditions  are  explored,  several  measures  -  both 
communication  and  performance  based  -  may  be  necessary  to  accurately  gauge 
congruence  across  multiple  contexts. 


Building  on  these  data,  then,  our  next  goal  will  be  to  use  these  findings  on  leading 
indicators  to  design  ways  to  support  and  then  empirically  explore  structural  adaptation. 
In  general,  we  believe  that  several  critical  conditions  will  have  to  be  met  before  a  team 
will  alter  its  organizational  structure,  ranging  from  training  to  cue  salience.  Thus,  as  one 
factor,  we  believe  that  by  using  leading  indicators  to  provide  organizations  with  some 
insight  into  how  their  structures  are  working,  we  may  be  able  to  help  induce  structural 
change.  This  supports  the  idea  of  a  model-based  “congruence  meter”  or  “fit  gauge”  that 
would  highlight  the  lack  of  congruence  at  any  point  in  time.  If  this  technological 
implementation  were  coupled  with  the  proper  training  indicating  that  changes  to 
organizational  structure  are  not  only  acceptable,  but  sometimes  required,  then  structural 
adaptation  may  occur  in  a  manner  that  can  support  mission  effectiveness. 
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Questions  &  Objectives 


1  Do  model-based  predictions  of  (in)  congruence 
produce  measurable  difference  in  process  and 
outcome? 


Organization 


Mission 


■  Measure  the  effects  of  congruence  on 
organizational  performance  and  processes 

■  Lay  the  foundation  for  further  work  on  structural 
adaptation 

■  Identify  leading  indicators  of  incongruence 

■  How  do  we  support/induce  adaptation? 


Overall  Performance  &  Process 
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Overall,  as  predicted  based  on  the  model  design  process,  relative 
to  the  congruent  conditions,  in  the  incongruent  conditions: 

•  Performance  was  worse 

*  Communications  volume  was  higher 
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The  Analysis  Goal 

>  Overall  results  indicated  that  the 
congruence  manipulation  was  successful. 

■  However,  to  successfully  support  structural 
adaptation,  we  must  identify  leading  indicators 
of  the  need  for  change. 

■  These  measures  must  be  identifiable  in  real 
time,  early  in  the  scenario. 

■  Thus,  focus  analyses  on  measures  of 
performance  and  process  over  time. 


“Congru-o-meter” 


Tempo 
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■  The  tempo  of  the  game  -  the  number  of  tasks  to  be 
processed  at  any  one  time  -  varied  over  time  and 
depended  on  condition 


Minimum 
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Maximum 
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Communications  Over  Time 


■  Differences  in  communications  volume 
persisted  over  time  and  were  present  early 


Tempo 


->  Bigger  changes  in  Functional 
Organization 


Communications  Over  Time 
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■  Differences  in  communications  about 
coordination  persisted  over  time  and  were 
present  early 
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->  Bigger  changes  in  Functional 
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Perceived  Workload  Over  Time 
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•  Workload  varied  over  time  and  depended  on 
condition 


■  Workload  tended  to  be  higher  in  incongruent  cases 


■  Diverges  with  time 


->  Large  drop-off  in 
Functional  Organization 
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Performance 


■  The  manipulations  of  congruence  were  successful 
in  changing  communications  and  perceived 
workload. 

•  These  changes  were  predicted  by  the  model-based 
manipulation  of  coordination  requirements. 

•  These  differences  were  present  early  in  the  missions. 

■  Given  these  changes  in  response  to  coordination 
needs,  we  expected  performance  to  be  worse  in 
the  incongruent  conditions. 

■  Will  performance  differences  be  present  early? 


Performance  Over  Time 
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*  Differences  in  the  frequency  of  attacks 


Task  Class 
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Incongruence  in  Action 


There  were  differences  in  communications, 
workload,  and  performance  between  the 
conditions  early  in  the  mission  scenarios. 

•  In  particular,  communications  differences  seemed  to 
be  present  early. 

Taken  together,  the  results  suggest  that  the 
Functional  and  Divisional  teams  adapted  their 
strategies  differently 

»  Compared  to  the  Divisional  teams,  in  response  to 
incongruence,  the  Functional  teams  changed  their 
communications  strategies  to  a  much  greater  extent. 

■  How  much  they  talked 

■  The  pattern  of  communication  (to  who) 

■  The  content  of  communication  (about  what) 
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Implications  for  the  Congru-o-meter 


»  In  the  context  of  this  experiment,  the  strategy  changes, 
or  leading  indicators  of  the  need  for  structural  adaptation, 
depended  on  structure-scenario  pairings. 

■  These  differences  mattered  even  in  these  “small”  and  “simple” 
organizations 

■  Strategies  for  coping  with  incongruence  may  differ  depending  on 
context  and  this  may  be  especially  true  for  complex  organizations 

■  Many  of  the  analyses  shown  here  are  calculable  in  real 
time,  as  demonstrated  by  the  over  time  analyses 

■  Communication  strategies  may  reflect  subtle  differences  and  are 
present  early  in  the  game 

•  It  may  be  possible  to  measure  communications  in  real  time  - 
likelihood,  frequency,  from/to,  and  potentially  even  content 

->  Is  this  the  road  to  a  congru-o-meter? 


Conclusions 
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■  The  organizations  and  scenarios  studied  here  set  the 
stage  for  further  work  on  structural  adaptation 

■  Based  on  modeling  work  we  successfully  created  the  conditions 
under  which  change  is  needed 

■  What  incongruence  looks  like  in  action  depends  on 
context 

■  Strategies  for  coping  varied 

■  The  leading  indicators  will  likely  be  complex 

■  Communication  strategies  in  response  to  incongruence  were 
different  in  D  and  F. 

■  Even  in  the  “small”  and  “simple”  organizations  studied  here 
contextual  effects  make  things 

■  The  congru-o-meter  will  need  to  be  context  sensitive 

•  Communication  measures  are  candidates  for  leading  indicators 


